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ABSTRACT 

The findings of the Cost, Quality, and Child Outcomes (CQO) 
study in Child Care Centers, the largest and most visible child care research 
conducted in the 1990s, were widely publicized and used to promote increased 
spending on initiatives to improve child care quality, the redesign of 
subsidy systems to provide quality incentives, more stringent licensing 
regulations, and overall increases in teacher compensation. This paper 
examines the methodology and execution of the study to determine whether they 
were robust enough to justify confidence in the study's findings. Scrutinized 
in this report are the sample design, response rates, methods of measuring 
child care costs, methods of measuring quality, and analytic issues related 
to the cost and quality component and the developmental outcomes component. A 
limited reanalysis of the cost and quality data is presented to shed 
additional light on problems in the study methodology. The report asserts 
that the combination of problems of selection bias, low response rates, and 
attrition makes it difficult to identify the population to which the findings 
might be generalized. Also questioned was the use of the center rather than 
the classroom as the unit of analysis, which, it is argued, ignores 
differences in cost and quality of child care for children of different ages. 
The report ends with two sets of conclusions: (1) the findings highlighted in 

the CQO report summaries overstated the findings of the analyses and do not 
identify the impact of improving center quality on child outcomes; and (2) 
given the flaws in the methodology and implementation of the CQO study, the 
study's findings should be interpreted with caution. (KB) 
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Introduction 



The Study of Cost, Quality and Child Outcomes in Child Care Centers, conducted by a consortium of 
respected academic researchers in the mid-1990s, was the largest and most visible piece of child care 
research conducted in the decade. Its results, and the conclusions drawn from those results, have been 
widely publicized and used by advocates to promote greatly increased spending on initiatives to 
improve child care quality, the redesign of public subsidy systems to provide incentives for higher- 
quality care, more stringent licensing regulations, and overall increases in the level of teacher 
compensation. 

While other arguments can be marshaled in support of many of these goals, it is important to 
scrutinize carefully the “scientific “evidence used to advocate for them. Most activities aimed at 
increasing child care quality have the concomitant effect of raising the cost of that care. When 
parents pay for care, the result may be to place some kinds of child care beyond their reach. When 
public fimds pay for child care, for a fixed level of resources, fewer children can be served. On the 
other hand, if a compelling case could be made that: a) care of a specified level of quality has a 
meaningful, (i.e., substantial and lasting) impact on the school readiness and performance of children 
who are otherwise at academic risk; and b) effective and reliable strategies exist to raise the quality of 
existing care to the level required to achieve such impacts, a compelling case could be made for 
vastly increased public investment. 

The Cost, Quality and Outcomes Study examined the relationship between the cost and quality of 
early childhood care and education programs and developmental outcomes for children enrolled in 
child care centers in four states: California, Colorado, Connecticut and North Carolina. The four 
states refiect the academic home of the researchers rather than a more systematic representation of the 
diversity of licensing standards across the United States; nevertheless, the four represent a range of 
stringency of licensing standards. Within each state, a stratified random sample of approximately 100 
centers was drawn from state licensing lists, with approximately equal representation of nonprofit and 
for-profit centers, all of which provided full-time care for infants and toddlers and/or preschoolers. 
Cost and quality data were collected during two-day visits to centers. Cost data were collected at the 
center level. A preschool classroom and an infant/toddler classroom (or two preschool or mixed 
classrooms, where no infant/toddler classrooms existed) were randomly selected to represent each 
program and observed. The center was then assigned a mean quality rating weighted by the number 
of classrooms of each type in the center. 
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A subset of the study centers, in which children could be observed for two years before entering 
school, was selected for the developmental outcomes component of the study. For this portion of the 
study, eligible centers were stratified by level of quality rating and economic status of children, 
randomly sorted, and then a quota sampling strategy was employed, i.e., researchers contacted centers 
in the order they were randomly sorted until the required number of children had been recruited and 
assessed. Developmental outcomes were assessed annually over a four-year period through 
individual child tests, teacher ratings and parent reports. Information on the family environment was 
gathered through parent interviews and on the school learning environment through observations. 

The study findings are set forth in the first public report (Cost, Quality and Child Outcomes Study 
Team, 1995) and, more recently, in The Children of the Cost, Quality and Outcomes Study Go to 
School (Peisner-Feinberg et al, 1999). The major findings can be summarized as follows: 

• Child care in most centers in the US is poor to mediocre. 

• Children’s cognitive and social development are positively related to the quality of their child 
care experience. 

• Quality of care is strongly related to staff-child ratios, staff education and administrators’ 
experience. Teacher wages, education and training also discriminate among centers of 
differing levels of quality. 

• Higher licensing standards are related to higher observed levels of quality. 

• Good quality services cost more but not a lot more. 

• Children who attended higher-quality child care centers demonstrated better cognitive and 
social skills from preschool into the early elementary school years. 

The primary purpose of this paper is to examine the methodology and execution of the study, in 
order to determine whether they were robust enough to give us confidence in the study’s findings. 

We also report briefly on a limited reanalysis of the cost and quality data that sheds additional light 
on problems in the study methodology. The reanalysis was limited for several reasons: first, we 
were not able to gain access to the outcomes data; second, the documentation of the cost data was 
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inadequate in some respects. Most importantly, concerns about the effect of low response rates on 
the generalizability of the data suggested that we limit the analyses we conducted. 

In the remaining sections of the paper we discuss issues related to the sample design and response 
rates; problems in the measurement and attribution of costs and quality; and analytic issues. The 
paper discusses what we learned from our limited reanalysis of the cost and quality data. The final 
sections of the paper discusses what we learned from our limited reanalysis of the cost and quality 
data. Two final sections briefly address the question, methodological issues aside, “Do the analytic 
results support the findings and recommendations set forth in the summary reports?” and present a 
set of conclusions. 

Sample Design and Response Rates 

Sample Design 

Cost and Quality Component. 

The CQO study used a multi-stage cluster sample design. In the first phase of the study (the cost and 
quality component), the first stage included a stratified random sample of 100 centers selected in each 
of the four states included in the study. Centers were stratified by for-profit and non-profit status, 
with approximately equal numbers of each included in the sample. In the second stage a stratified 
random sample of two classrooms was selected in each of the centers included in the sample. 
Classrooms were stratified by the age of the children served (less than 30 months, and 30 months and 
older). Classrooms serving school-age children were excluded from the sample. If a center included 
both infant/toddler classrooms and preschool classrooms, then one room was randomly selected from 
each stratum. If a center did not serve infants or toddlers, then two preschool classrooms were 
randomly selected. 

The final sample for the cost and quality component included a total of 401 centers (201 for-profit 
and 200 non-profit), and 749 classrooms* (228 infant/toddler and 521 preschool). For center-level 
analyses, this sample provides a level of precision of D.05 (P=.50, • =.05), and statistical power of .80 
to detect differences of about .25 standard deviations between for profit and non-profit centers. For 
classroom-level analyses, the sample provides a level of precision of D.07 for infant/toddler rooms 
and D.04 for preschool rooms. 



‘ There were 5 1 centers with only one classroom. 
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Developmental Outcomes Component 



The developmental outcomes component was a longitudinal study of child outcomes. The sample of 
children for the developmental outcomes component of the study was nested within the classrooms in 
which quality was assessed. Each of the 401 centers included in the cost and quality component of 
the study was asked to participate in the developmental outcomes component. Only 45 percent (181) 
of the centers agreed to participate. Of these 181 centers, 151 were included in the developmental 
outcomes components. No reason was given for the exclusion of 30 centers that agreed to participate. 
A sample of 183 was selected from the preschool classrooms where quality was assessed during the 
first phase of the study. Eligible children included children who were expected to enter kindergarten 
in fall 1994, were enrolled in the target classroom during the quality assessment period, and who were 
expected to attend the same center the following year (Peisner-Feinberg, et al., 2000). 

The final sample for the developmental outcomes component included a total initial sample of 745 
preschoolers nested within 183 classrooms in 151 centers.^ Outcomes for these children were 
measured in study years 1, 2, 3, and 5 (the last two years of preschool, kindergarten, and second 
grade). With attrition from the sample, only 352 children remained in the sample in year 5.^ 

However, because of the clustering of children within centers, the effective sample sizes are 
considerably smaller. We estimate that the effective sample sizes are approximately 414 children in 
year 1 and only 196 children in year 5."^ These sample sizes are too small to support the types of 
analyses conducted in the CQO. This issue is discussed in more detail below in the context of the 
imit of analysis and analytic techniques used by the CQO. 

Apart from the adequacy of the sample sizes, the major problem with the CQO sample design is that 
the sample of children is limited to children who were expected to be in the same center in year 2 



^ The authors report an initial sample of 826 children. However, the maximum sample size used in the 
analysis is 745 children. Incomplete data were available for the remaining 81 children recruited (Peisner- 
Feinberg, et al., 2000). 

^ The authors do not report the number of centers or classrooms remaining in the sample in year 5. 

The effective sample size is equal to the actual sample size (n) divided by the design effect (dejf)> where\ 
dejff^\-^[* (b-I)], and 
• =intraclass correlation 
b=average number of children per center. 

The initial CQO sample has an average of 5 children per center. Since centers are likely to group children 
relatively homogeneously, we have assumed a relatively high intraclass correlation of* =.2. Under these 
assumptions, deff=^\.%. 
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(Helbum et el. 1995, p. 38; Peisner-Feinberg, 1999, p. 3). The CQO study notes that this was 
intended to limit the sample to children “who had relatively stable child care situations.” Since one 
would expect stability of child care arrangements to be influenced by many of the same factors that 
effect child outcomes, (e.g., household social economic status), the systematic exclusion of children 
of children with less stable child care arrangements introduces some degree of bias into the study 
estimates. The CQO does not provide any information about the number of children who were 
excluded from the sample because they did not return to the same center in year 2. Had the CQO 
included a small subsample of children who had a different child care arrangement in year 2, it would 
have been possible to estimate the magnitude of this bias. Unfortunately, this was not done in the 
CQO and one can only speculate as to the potential magnitude of this bias. 

Response Rates 

The response rates in the CQO study are quite low, raising the issue of potential nonresponse bias in 
the study estimates. In the cost and quality component, only 52 percent of eligible centers sampled 
agreed to participate in the study (Helbum, et al., 1995). Because the cost and quality of the centers 
that refused to participate in the study is unkown, this response rate is low enough to raise issues of 
potential bias. Had the CQO study conducted a study of cost and quality in a small subsample of the 
centers that initially refused to participate (perhaps by providing them with a monetary incentive 
sufficient to gain their cooperation), the magnitude of the potential bias could have been estimated. 
Again, this was not done and one can only speculate about the potential magnitude of the bias in the 
study estimates. 

While the CQO study provides information on the response rate for the cost and quality component, it 
does not provide this information for the developmental outcomes component. The cumulative 
response in a multi-stage sample is equal to the product of the response rates for each stage of the 
design. The developmental outcomes component includes three sample stages: 

Stage 1 : Centers for the cost and quality component 

Stage 2: Centers in the cost and quality component that participate in the developmental 

outcomes component 

Stage 3: Parents in participating centers. 

As noted above the Stage 1 response rate is 52 percent. The Stage 2 response rate is 45 percent (181 
centers out of the 401 centers in the cost and quality component agreed to participate in the 
developmental outcomes component). The Stage 3 response rate was not reported. A total of 961 
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parent consent forms were returned, but no information is provided on the total number of these forms 
that were distributed to parents. However, if we assume that half of the children in each sample 
classroom are eligible to participate in the study, we estimate that the Stage 3 response rate was 
approximately 64 percent [based on the reported average number of classrooms per center (1.2) 
included in the sample, and the reported average class size (13.7).] The estimated cumulative 
response rate for the developmental outcomes component is only 15 percent. And this is before 
attrition from the sample. With such a low response rate and no evidence that non-responders are not 
meaningfully different from responders, one should treat the results of this component of the CQO 
study with considerable caution. 

Measuring Costs 

The protocol used by the CQO study to collect data on the costs of child care centers is the most 
comprehensive instrument that has been used to date by researchers trying to understand the costs of 
operating early childhood programs. Full-cost accounting requires that the cost of all resources used 
by a child care center be identified and attributed to center operations. As noted in the CQO study, 
these include costs incurred by a center and reported on its statement of income and expense as well 
as the value of in-kind contributions (e.g., volunteer labor and donated space). While the CQO study 
did an outstanding job of identifying in-kind contributions, their methodology for assigning a value to 
these in-kind contributions results in an over-estimate of the real value of these contributions to the 
center. In addition, including the value of foregone wages in their full-cost estimates results in a 
substantial over-estimate of costs. On the other hand, the CQO study may have under-estimated 
costs for child care centers that are part of a larger organization. 

Assigning a Value to In-kind Contributions 

The CQO study collected detailed information on the in-kind contributions received by child care 
centers and the prevailing market rates for the resources received. For example, if a center operated 
in donated space (e.g., a church basement), information was collected on the amount of space and the 
prevailing cost per square foot of similar space in the vicinity. The problem is that the CQO study did 
not apply a market test to the in-kind contributions. While the study asked centers what they would 
have had to pay for the in-kind contributions if they purchased them, the study did not ask if the 
centers would have purchased these resources had they not been donated. For example, a center may 
be given the use of a 4,000 SQFT church basement with a rental value of S4.00/SQFT. Suppose that 
this center has a planned enrollment of 35 FTE children. This works out to 1 14 SQFT/FTE child. 
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The CQO study would have assigned a value of $16,000 (i.e., 4,000 SQFT @ S4.00/SQFT) to this 
contribution. However, before assigning a value to this donated space, it is necessary to ask whether 
or not the center would have purchased (or rented) this much space had it not been donated. If, in the 
absence of donated space, the center would have only used 60 SQFT/FTE child, then including the 
value of the entire 4,000 SQFT substantially overstates the real value of the in-kind contribution to 
the center. Applying a market test to the contribution, we would only include the value of the amount 
of space the center would have purchased (or rented). The proper imputed value of the donated space 
is $8,400 [i.e., (35 FTE children) (60 SQFT/FTE child) ($4.00/SQFT). Unfortunately, it is not 
possible to re-analyze the CQO study dataset applying a market test to the in-kind contributions 
received. 

Centers that Are Part of a Larger Organization 

Many child care centers are part of a larger organization. For profit centers run by chains are but one 
example; in the nonprofit sector, centers are often one component of a multi-service agency (e.g., 
centers run by the YMCA or Catholic Charities). As a unit within a larger organization, such a center 
may be viewed as a cost center for which the management of the larger organization chooses to 
accumulate and report operating costs. As noted above, full-cost accounting requires that we include 
the cost of all resources used by a center. This includes costs incurred by the larger organization in 
support of center operations. These latter costs may or may not be charged to the center. 

In some cases, these costs may be captured by an indirect cost rate and included in the costs reported 
on a center’s statement of income and expense. In other cases, centers may be charged a fee-for- 
service for the support received from the larger organization, in which case they will be reported on a 
center’s statement of income and expense. However, in many cases, the larger organization makes no 
attempt to capture these costs, in which case they are not reported on a center’s statement of income 
and expense. The CQO study included costs incurred by the larger organization only to the extent 
that a center was charged (either directly or indirectly) for these support services. If a parent 
organization did not charge a center for these services, the CQO study under estimated the center’s 
full cost. Unfortunately, it is not possible to identify and estimate the value of these unreported costs 
for child care centers using the CQO study dataset. 

Foregone Wages 

In calculating the full cost of care, the CQO study included the imputed value of foregone wages of 
center staff. The argument made for including foregone wages in the full-cost estimate is ‘that center 
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staff subsidize the cost of care through their low wages’ (Helbum, 1995). While it is certainly true 
that child care workers are paid low wages, the argument for including the value of their foregone 
wages in the full-cost estimate is spurious. As discussed above, the value of in-kind contributions 
should be subject to a market test — ^how much of these resources would the center have used if it had 
to pay for them. The wages paid to child care staff are determined by the local labor market, and by 
definition are subject to a market test. The supply of caregivers reflects their willingness to work in 
child care at the prevailing market wage. If child care workers had other (higher-paying) options that 
they preferred, they would not be working in child care at the prevailing wage rate for caregivers. 

That caregivers are working in child care indicates that either they don’t have other higher-paying 
options, or their preference for child care work is worth more to them than the additional wages that 
they could receive in other employment. In either case, their foregone wages should not be 
considered as a cost of providing child care. 

The inclusion of foregone wages has a major effect on the full-cost estimates. The value of foregone 
wages accounted for about half of the difference between total expended cost and total full cost. 
However, since foregone wages are clearly identified in the CQO study dataset, it is possible to 
improve the estimates of the full cost of care by excluding foregone wages. 

Measuring Costs at the Center or Classroom Level 

The Cost and Quality Study (as all previous studies of child care center costs) measured costs at the 
center level. While this is a fine approach for analyzing the cost of center operations (e.g., estimating 
the average cost per child-hour of care provided, and examining the components of center costs), it is 
problematic for examining the relationship between costs and quality. Quality is measured at the 
classroom level, not the center level. In the Cost and Quality Study two classrooms were randomly 
selected in each center included in the study. A weighted average of the quality measures for these 
classrooms was used as an estimate of the overall level of quality in the center (Morris & Helbum, 
1997). The problem with this approach is that the center-level measure of cost may not reflect the 
costs attributable to the target classrooms that were used to construct the center-level quality measure. 
This may mask, or distort, the estimated relationship between cost and quality. Quite simply, both 
quality and cost should be measured at the classroom level and the level of analysis for examining the 
relationship between cost and quality should be the classroom not the center. 

Approach for measuring cost at the classroom leveL Commonly used in business, a system of 
functional accounts is designed to provide management with information regarding the cost of each of 
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the organization’s activities. In general, an organization’s activities fall into two broad categories: 1) 
the production of end products for sale to customers, and 2) management and service activities that 
support the production of end products. A system of functional accounts also facilitates the 
determination of the full cost of each of the end products. 

The cost of management and service activities are viewed as indirect product costs in that they are 
not traceable to specific end products. In full-cost accounting, a share of each of the management and 
other service activities is allocated to each of the organization’s end products. For example, in 
manufacturing it is common to distribute management and plant supervision costs in proportion to the 
number of employees engaged in the production of each product, while distributing maintenance 
costs in proportion to the value of the equipment used in the production of each product.^ 

In many respects, a child care center is no different from any other organization engaged in the 
production of several different end products. Despite the variation among child care centers in size, 
ages of children served, hours of operation, and organizational complexity, the major activities that 
form the basis of a system of functional accounts are relatively simple. The major activities are: 1) 
direct provision of child care, and 2) non-child care activities (including center administration, space, 
transportation, and food). In a functional cost analysis, all non-child care costs are seen as activities 
in support of the provision of child care. In this context, each classroom in a child care center may be 
viewed as a cost center (which is analogous to an end product). The cost of non-child care activities 
must be distributed to each classroom. The full cost attributable to each classroom includes its direct 
child care costs plus a share of the cost of non-child care activities. 

The distribution of line-item cost elements to functional activities involves a combination of direct 
attribution and allocation. Labor costs are directly attributable to child care and non-child care 
activities. Salary schedules and work assignments may be used to identify the labor costs associated 
with each classroom and non-child care activities. Costs other than labor must be allocated across all 
of the classrooms in a center. There are several bases available for this allocation: 

• the proportion of direct child care labor costs attributable to each classroom; 



^ The concept of functional cost accounting and the issues involved in allocating the cost of management and 
other support activities to end products is contained in standard texts on managerial accounting. See 
Shillinglaw, Gordon and Phillip Meyer, Accounting: A Management Approach (Homewood, IL: Richard D. 
Irwin, Inc., 1983) 
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• the proportion of FTE caregivers assigned to each classroom; 



• the proportion of FTE children enrolled in each classroom; and 

• the proportion of total licensed capacity attributable to each classroom. 

Labor cost and labor hours (i.e., FTE caregivers) are the predominant methods used by business. 
However, in a child care center, using FTE children is a viable alternative basis for this allocation. 
Since licensed capacity is, for the most part, determined by the amount of useable space in a center, 
using licensed capacity as the basis for the allocation is tantamount to allocating non-child care costs 
on the basis of the physical size of each classroom (a method that does not have much intuitive appeal 
in a labor intensive business such as child care). 

The choice between using labor (either cost or FTEs) or FTE children is somewhat arbitrary. 
However, using labor as the basis for the allocation will have the effect of allocating a higher 
proportion of non-child care costs to classrooms with younger children, especially infant/toddler 
rooms, because of the higher staff/child ratios required for younger children. Conceptually, the 
question boils down to whether the use of administrative resources is more closely associated with 
dealing with staff or dealing with children/parents. Similarly, the use of labor cost will be sensitive to 
the staffing configuration of classrooms (i.e., for any given number of FTE caregivers, classrooms 
using more teachers and fewer aides/assistants would have higher labor costs). The decision about 
which method to use for the cost analyses should be based on a test of the sensitivity of the cost 
estimates to the allocation method used. If the estimates prove to be sensitive to the allocation 
method used, then it is necessary to present the full range of the cost estimates. 

Measuring Quality 

The set of measures used to rate the quality of the child care environment was comprehensive enough 
to allow the researchers to assess several different aspects of that environment. The ECERS (and its 
companion measure, the ITERS) is a 37-item scale that allows the observer to rate seven aspects of 
the environment. Each item is scored on a seven-point scale from inadequate to excellent, and a total 
score is computed to give an environmental rating. These seven subscales vary in their likely 
importance for the child’s development, but cany the same weight in the overall score. For example, 
the four items that measure the extent to which the environment meets adult needs make the same 
contribution to the overall score as the four items items that assess the quality of children’s language 
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and reasoning experiences. Nevertheless, the measure is widely used and highly regarded. 
Increasingly, it is used by states and localities, not only or even primarily as an evaluation tool but, as 
it was originally intended, as a diagnostic tool and a basis for prompting programs to work on 
improving the quality of the environment for children and adults. 

The second measure, the Caregiver Interaction Scale (Arnett, 1989), fills a gap left by the ECERS and 
assesses the affective dimension of the caregiving provided. Using the scale, the observer rates the 
sensitivity, harshness, degree of attachment and permissiveness of the lead teacher.^ The third 
measure, the Teacher Involvement Scale (Howes and Stewart, 1987), measures the amount and 
quality of the interactions between the teacher and the children in her care. Again, there is no 
indication that this measure was used on any but the lead teacher in each classroom (and only one 
subscale from this measure was used to compute the final overall score). A fourth observational 
measure was used to assess aspects of the school classroom environment, for those children who were 
followed into elementary school. 

For the cost and quality component of the study, results from these three measures were combined in 
a principal components analysis to produce a single score for each of two the classrooms observed. A 
weighted average^ of these two scores was then computed to create an overall index of the quality of 
the center environment. 

The use of three different measures of environmental quality is admirable, and allows comparisons 
with several other studies. However, the ways in which scores from the measures were combined, 
analyzed and reported raise some issues. The combining of scores from two classrooms to create a 
center-level score would be a reasonable strategy if all the centers assessed had only preschool 
classrooms. But, in this study, some programs had infant and toddler classrooms, others had only 
preschool classrooms. Infant-toddler care presents more challenges than care for preschoolers. 

Center directors are under great pressure to provide care for infants and toddlers, as well as 
preschoolers. At the same time, the licensing standards for infant and toddler care are considerably 
more stringent than for preschool care, even in states with relatively lax requirements. For example. 
North Carolina at the time of the study allowed ratios of 1:18 or 1:20 for preschool classrooms, but 
required ratios of 1:6 for infants and toddlers. This could mean that infant-toddler care in the state 

^ In some studies, both the lead teacher and the assistant teacher are rated and the ratings are then combined 
and averaged. In this study, only the lead teacher was rated. 

’ Scores were weighted by the number of preschool and infant and toddler classrooms in the center that each 
score represented. 
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was three times as expensive as preschool care, but that is usually not the case. If, instead, directors 
must try to cut comers to provide care that is somewhat affordable, it is likely that infant-toddler 
classrooms demonstrate wider variation in quality and, on average, are of lower quality. If either or 
both of these statements holds true in the CQO study, then the average quality rating of centers that 
contained both types of classrooms would automatically be lower than the average for centers that 
only had preschool classrooms. That is, the major factor responsible for the difference in quality 
would be the existence of an infant/toddler classroom. In the reanalysis of the cost/quality data that is 
reported later in the paper, we examined these questions. Our findings support the position that 
quality, however measured, should have been analyzed and linked to costs at the classroom level. 

It is also important to add a note of caution about the statements made, mostly on the basis of the 
ECERS and ITERS, about the distribution of quality scores. The scores on the ECERS and ITERS 
are grouped in three categories - poor (average scores from 1 to less than 3), mediocre (average 
scores from 3 to less than five) and developmentally appropriate), but these are arbitrary divisions and 
cut-off points. Indeed the researchers themselves, though they describe and define each of these 
categories in detail, actually use a different grouping of scores in their later analyses, without 
discussion of their reasons (Peisner-Feinberg et al, 1999, p26). 

The two reports (Cost, Quality and Outcomes Team, 1995; Peisner-Feinberg, 1999) describe the same 
analytic results in radically different terms. The first report states, as its first major finding, “Child 
care at most centers in the United States is poor to mediocre.” The technical report, that accompanies 
it notes that the overall mean of 4.0 is a full point below good quality (Helbrun, 1995, p.26)). By 
contrast, the 1999 report, discussing the same set of analyses, states “ the observed quality of child 
care experienced by children in the study (our italics) tended to be medium” (Peisner-Feinberg, 1999, 
p.8). What exactly does medium mean? The report goes on to elaborate. The average global 
classroom environmental quality score was 4.26 in year 1 and 4.05 in year 2 of preschool on the 
ECERS) “Teachers were rated as moderately sensitive (2.96 on a 4-point scale) were observed to be 
at least minimally responsive to a target child an average of 31% of the time. Children engaged in 

interactive play with peers an average of 77% of the time On average, the teaching approach 

observed in the classroom was slightly closer to a child-centered approach than a didactic approach 
(3.49 on a 1 to 5 scale).... teacher-child relationships tended to be positive, with teachers reporting 
relatively close relationships (mean of 4. 17 for year 1 and 4. 14 for year 2 on a 5-point scale) that were 
generally not overdependent or conflicted” (Peisner-Feinberg, 1999, p.8). In spite of the careful 
modifiers, this paints a picture different from and rosier than the first report, though both derive from 
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the same set of analytic results. Would it have generated the same level of attention as the statement 
in the first report? 

If there is still confusion about what a score of 4 on a 7-point scale represents, here are some 
examples from the ECERS. A classroom would receive a score of 3.0 (minimal) if it had some 
appropriate gross motor equipment, but it was seldom in use, and a score of 5.0 (good) if gross motor 
equipment was “readily available and sturdy, stimulating a variety of skills, and if building and 
dramatic play equipment were included in gross motor areas.” If one or a part of one of these 
statements could not be made, e.g., if dramatic play equipment was not included in the gross motor 
area, then the classroom would receive a score of 4. On an item assessing opportunities for teachers’ 
professional growth, a center (this is not a classroom-level item) would receive a “minimal”rating if it 
had a limited professional library, staff meetings were limited to administrative issues and no in- 
service training was provided, and a “good” score if it had a good professional library, with current 
materials on a wide variety of subjects readily available, as well as regular staff meetings which 
include staff development activities as well as a plan for orienting new staff. A center with a limited 
professional library but all the other components of the “5” rating, would score 4 on this item. 

These examples make one aware of how the ECERS can be used to sharpen teachers’ and center 
directors’ awareness of good practices. However, it is an open question whether the distinctions 
between points in the upper half of the scale are meaningful for policy research. 

Analytic Issues 

Cost and Quality Component 

A significant issue in examining the relationship between cost and quality in early childhood 
programs is determining the appropriate unit of analysis. Typically, cost is measured at the center 
level, and this was the approach taken by the CQO study. However, quality is inherently a classroom 
variable, and the CQO study measured quality in a sample of two classrooms in each center. The 
issue then, is how to relate center-level costs to the quality of individual classrooms. The approach 
taken in the CQO study was to conduct the analysis using the center as the unit of analysis — the 
average quality of the two sample classrooms was taken as a measure of the overall quality of the 
entire center. This approach is seriously flawed. Costs clearly vary considerably from classroom to 
classroom — staff/child ratios vary across different age groups (and often within age groups), and the 
amount of space and other resources varies across classrooms within a center. Indeed, an often cited 
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reason for not serving infants is that the cost of providing care to infants is considerably higher than 
the cost of serving preschool and school-age children. 

Developmental Outcomes Component 

There are two major problems with the developmental outcomes component of the study. The first, 
and most serious, is that the CQO study (in common with many previous studies of the relationship 
between child care quality and developmental outcomes) does not adequately control for the factors 
that influence parents’ selection of child care arrangements for their children. Without adequate 
controls for the selection of children into child care arrangements, the CQO study estimates of the 
effects of child care quality on developmental outcomes are subject to selection bias. As Blau (1999) 
has noted, “parents who provide a home environment that fosters positive child outcomes would 
plausibly select child care arrangements that do so as well.” 

The CQO study collected a considerable amount of information on family characteristics, including 
both demographics and family practices and beliefs. The intent was to use this information to address 
the critical issue of self-selection. “The parental beliefs and family environment information provides 
further description of home environments in our sample and allows us to explore in more detail the 
relations between family characteristics to selection of child care and child development” (Piesner- 
Fienberg, et al., 1999, p.l4). There is no evidence presented in the CQO study report that indicates 
that such analyses were conducted. Only three background variables were included in the analyses of 
developmental outcomes — maternal education, child’s gender, and race. Using CQO study data, Blau 
(1998) has shown that including more family characteristics in a model explaining the quality of child 
care arrangements selected increases the explanatory power of the model (i.e., the proportion of the 
variance explained), but greatly reduces the sample size. He also notes that complete information was 
less likely to be available for children itom disadvantaged households in lower quality child care 
centers. Despite the availability of information that could help explain child care selection, the CQO 
study limited their analysis of factors related to child care selection to maternal education, gender, and 
race , in part, “because these variables were available on almost all of the children in the sample 
(Piesner-Fienberg, et al., 1999, p.l5).” It would have been helpful if the CQO study had conducted 
(and reported) parallel analyses using the more limited sample for which complete information on 
parental beliefs and family environment were available. In the absence of information to show that 
self-selection issues have been adequately addressed in the analysis, it is difficult to attach much 
credibility to the findings relating child care quality to developmental outcomes. 
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The second major analytic issue is the treatment of attrition from the sample. As noted above, there 
was a significant amount of attrition from the sample between year 1 and year 5. If those children 
that leave the study are different from those who remain in the study in ways that are related to 
developmental outcomes, then the study estimates of the relationship between child care quality will 
be biased. There is no indication that the CQO study examined the potential magnitude of such bias. 
Again, in the absence of information to show that the effects of selective attrition have been 
adequately addressed in the analysis, it is difficult to attach much credibility to the findings relating 
child care quality to developmental outcomes. 

A Limited Re-Analysis of the CQO Study Data 

The CQO study provided Abt Associates with the data files from the cost and quality component of 
the study. A limited re-analysis of these data was conducted to a) test the feasibility of measuring 
cost at the classroom level, and b) to examine the efficacy of conducting the cost and quality analyses 
at the classroom level rather than at the center level as was done in the CQO study. 



Classroom Costs 

Using the procedures described above, we calculated the full cost for each of the classrooms included 
in the study. Table 1 presents a simple comparison of the cost per child in infant/toddler and 
preschool classrooms. As would be expected, the average annual cost per FTE child is nearly 50 
percent higher in infant/toddler classrooms ($6,422) than in preschool classrooms ($4,335). 



Table 1: Comparison of the Cost and Quality in Infant/Toddler and Preschool 
Classrooms 




ITERS/ECERS 


Annual Cost/FTE Child 




Infant/Toddler 


Preschool 


Infant/Toddler 


Preschool 


Mean 


3.5 


4.3 


$6,422 


$4,335 


Median 


3.3 


4.3 


5,851 


3,707 


Std. Dev. 


1.1 


1.0 


3,110 


2,433 


CV 


20.7 


23.1 


48.4 


56.1 


N 


206 


463 


216 


472 



This difference is statistically significant (P<.0001). This cost differential lends support to the 
argument that the cost and quality analyses should have been conducted at the classroom level rather 
than at the center level. 
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Classroom Quality 



The CQO study measured quality at the classroom level and then created a center-level measure of 
quality by taking the weighted average of the quality scores in the two sample classrooms.^ While the 
CQO study examined several measures of quality, and developed an overall composite measure of 
quality based on the ECERS/ITERS, Arnett, and Howes Responsiveness measures, most of their 
discussion of quality is framed in terms of the ECERS/ITERS measure.^ For this reason, we focused 
our limited re-analysis of quality on the ECERS/ITERS measure. 

A comparison of the ECERS/ITERS of infant/toddler and preschool classrooms is shown in Table 1. 
Two things stand out in this comparison: a) the quality of infant/toddler classrooms is significantly 
lower than that of preschool classrooms (a mean ECERS/ITERS of 3.3 vs. 4.3); and b) quality is 
considerably more variable in infant/toddler classrooms than in preschool classrooms (CV of 30.7 vs. 
23.1).*® Given these differences, it is quite possible that the CQO study’s center-level measure of 
quality is biased against centers that serve infants and toddlers. That is, because of the manner in 
which the center-level quality measure is constructed, we would expect that the center-level measure 
of quality would be lower in centers that serve infants and toddlers than in centers that only serve 
preschool children. To test this hypothesis we compared the center-level ECERS/ITERS in centers 
that do, and do not, serve infants and toddlers (Table 2). The ECERS/ITERS is significantly higher 
for centers that do not serve infants and toddlers (P<.0001) and is very similar to the difference 
observed between the classroom ECERS/ITERS measures.** This also lends support to the argument 
that the cost and quality analysis should have been conducted at the classroom level. 



Table 2: Comparison of Center-Level ECERS/ITERS Scores for Centers with 


Infants/Toddlers and Centers with Oniy Preschooi Chiidren 




Type of Center 


Mean 


Std. Dev. 


N 


Has infant/Toddlers 


3.7 


0.89 


219 


No infants/Toddlers 


4.3 


0.85 


179 



® The weights were based on the proportion of center enrollment in each of the two age groups. 

’ For example. The CQO study definition of high, medium, and low-quality centers is based on the seven- 

point ECERS/ITERS scale: less than 3 = low quality; 3-5 = medium quality; and 5-7 = high quality. 

The coefficient of variation (CV) is a measure of the relative variation. The CV = 100(Std. Dev./Mean). 

* * We also compared the center composite measure of quality with similar results. Centers that serve 
infants/toddlers had significantly lower scores on the center-level composite measure of process quality 
than centers that only serve preschool children. 
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Relationship Between Cost and Quaiity 



A simple linear regression analysis was conducted to assess the relationship between cost and quality 
at the classroom level, controlling for state and classroom type (Table 3). A statistically significant 
relationship of very low magnitude is observed between cost and quality. This relationship is 
different in North Carolina relative to the other three states. In California, Connecticut and Colorado, 
a $100 difference in the annual full-cost per child is associated with a positive difference of 0.010 on 
the ECERS/ITERS scale. However, in North Carolina a $100 difference in cost per child is 
associated with a positive difference of 0.020 on the ECERS/ITERS scale. 

Of note is that the quality of infant classrooms is significantly lower than that of preschool 
classrooms, even after controlling for the differences in cost. One possible explanation for this is that 
centers use different types of staff in infant/toddler classrooms and preschool classrooms. For 
example, staff in infant /toddler classrooms had less formal education, less prior experience in child 
care, and less tenure in the center than staff in preschool classrooms. 



Clearly, more work needs to be done in this area before we begin basing policy decisions on the 
purported relationship between cost and quality. However, if the above findings are any indication, 
we need to look beyond simply calling for higher expenditures on child care 



Table 3: Relationship Between Cost and Quality at the Classroom Level (n=666) 


Variable 


Regression 

Coefficient 


Standard Error 


t Value 


Annual Full Cost Per Child 


0.000096 


0.000019 


35.66 


Colorado 


-0.19519 


0.101.83 


-1.92 


Connecticut 


-0.11790 


0.10386 


-1.14 


North Carolina 


-0.93020 


0.17722 


-5.25 


Infant Classroom 


-1.04138 


0.17661 


-5.90 


Cost X North Carolina 


0.000105 


0.00003 


3.15 


Cost X Infant Classroom 


0.000011 


0.00003 


0.69 
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Do the Results of the Analyses Support the 
Findings and Recommendations? 



The preceding sections of the paper have examined the strengths and weaknesses of the study design 
and its implementation. We now turn to the question of whether the results of the analyses that were 
carried out support the major findings and recommendations of the study. 

The first widely-cited finding in the 1995 report states that ""child care in most centers in the US is 
poor to mediocre” (Cost, Quality and Child Outcomes Team, 1995, p.26) As we noted earlier in the 
paper, the second report provides a more positive and more carefully phrased statement of the overall 
results of the quality analyses. It correctly limits the generalizability of the finding to the experience 
of children in the study. The results of these analyses cannot be generalized to centers in the US, 
since the states were purposively chosen and the distribution of the center sample in terms of the 
proportion that served infants and toddlers does not accurately reflect the distribution nationally. 
Center-based infant-toddler care is even rarer nationally than it is in this sample where less than a 
third of the centers had infant-toddler classrooms. In this sample, mean quality scores for centers 
were driven downward in those centers that had infant-toddler classrooms, because the quality of 
these classrooms was lower and more variable than the quality of their preschool classrooms. A more 
accurate statement of the results of the quality analyses would be that for almost 60% of the preschool 
classrooms in the study, the quality of the environment was medium to excellent (scores of 4.0 to 
7.0), but that the proportion of infant-toddler classrooms that passed this threshold of quality was 
lower (32%). This would appropriately focus attention on the reasons for the disparity in quality 
scores and the problems that might be encountered in attempting to improve quality in center-based 
care for infants and toddlers. 

The second major finding of the study, that “children’s cognitive and social development are 
positively related to the quality of their child care experience” replicates the findings of many 
other studies (many of which were conducted by members of the CQO study team). Because there 
are no studies that have tested this proposition experimentally, most studies address the issue of 
possible selection bias by controlling for a set of family background variables, a strategy that this 
study also used. The problem is that, in addition to the variables that we can measure and control for 
(in this study, maternal education and the ethnicity and gender of the child were used as control 
variables), there exist a host of other unmeasured variables that indicate pre-existing group 
differences and that could influence parental selection of child care arrangements. If quality in 



Abt Associates Inc. The Study of Cost, Quality and Child Outcomes in Child Care Centers 



18 



church-based centers is systematically lower, as the study found, what is it that leads parents to select 
this type of care? 

We would conclude that the finding reasonably reflects the results of the analyses and should be read 
with the caveats about likely selection bias and, in this study, the added problems of response rates 
and attrition. 

The third major study finding is that “the quality of chUd care is positively related to higher staff- 
child ratios, staff education, and administrators’ experience”. Teacher wages and specialized 
training also discriminated among centers with different levels of quality. What is not highlighted 
here is that the regression analyses managed to account for only half the variance in quality among 
centers. “Much of the difference in quality among centers was left unexplained” (Helbum, 1995, 
p.294). The authors of this portion of the report go on to suggest that what the results of their 
analyses call for is additional research on other aspects of child care quality. Another intriguing result 
of the analyses not highlighted by the authors is that teacher wages were only related to the quality of 
care when an average wage was calculated for all teaching staff and used in one set of analyses. In 
the econometric analyses, when wages for teachers with baccalaureates, with some college and with 
no formal education beyond high school were examined separately, only wage levels for staff with 
the lowest level of formal education were significantly related to quality. For teachers with college 
degrees, differences in wages did not affect the quality of the classroom environment. 

Another widely-cited finding is that good quality care costs more, but not much more. The study 
estimates that it would cost an additional 16 cents per hour per child to raise the quality of the 
environment from medium (4.0) to good (5.0). As we noted earlier, we believe this analysis should 
have been conducted at the classroom level, differentiating between infant-toddler and preschool 
classrooms, because of the wide differences in costs and quality between the two types of classroom. 
In addition, and presupposing that we have reliable strategies for increasing quality between these two 
points at either the center or the classroom level, we are not told what this particular increase in 
quality buys us in terms of enhanced child development. In the second report, focusing on the 
longitudinal analyses of child outcomes, differences in outcomes are examined as a function of two 
extremes of quality: centers in the top 25% of quality vs. centers in the lowest 25% of quality ratings. 

The last finding that is worth commenting on is the major finding in the second report that children 
who attended higher-quality centers demonstrated better cognitive and social skills from 
preschool into the early elementary years. Two points are worth making here. The first is that the 
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comparisons that are highlighted are between children who attended the highest quality centers and 
children who attended the lowest quality centers, although the analyses probably used quality as a 
continuous variable. We would like to have seen how children who attended “medium-quality” 
centers performed compared with children in “good-quality” centers, since those are the points that 
the cost-quality analysis focused on. Even with these extreme comparisons, the results are not very 
impressive. By second grade, most of the differences have disappeared. Of the six outcome 
measures used, there are effects of child care quality on two - math skills, and one of two social- 
emotional measures. The effect on the math score is between .1 and .2 of a standard deviation, an 
effect that is not considered educationally significant. We are not given enough information about the 
social-emotional measure to assess the importance of the effect. Finally, at this point, there has been 
additional and substantial attrition from the sample. 

Conclusions 

Our examination of the both the presentation of the study findings and the methodology and 
implementation of the design and analyses used to generate the findings leads us to two sets of 
conclusions. 

First, the researchers, in particular the econometricians, carried out a series of careful analyses. 

Setting aside our arguments about the analytic strategies used and the appropriate level of analysis, 
we believe that the findings that are highlighted in the summaries of both reports overstate the results 
of these analyses. Further, after asserting that a substantial change in the average quality of centers 
could be effected for little additional expenditure, the researchers fail to identify for us what, if any 
impact this change would have on either short-term or long-term outcomes for children. 

Second, we believe that the flaws we have identified in the methodology and implementation of the 
study should lead us to view the results of the analyses with caution. The combination of problems of 
selection bias, low response rates and attrition makes it difficult to identify the population to which 
these findings might be generalized. The use of the center rather than the classroom as the unit of 
analysis, which ignores the differences in cost and quality of child care for children of different ages, 
introduces additional problems. 

Michael Lamb, perhaps the most clear-sighted of the researchers who have examined the 
relationships between child care quality and child outcomes, suggests that, while out-of-home care of 
superior quality is beneficial to children and preferable to care of poor quality, it is essential to 
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evaluate the relative costs and benefits of incremental improvements in quality, rather than to view 
“quality” as a magic bullet (Lamb, 1992). The CQO study researchers have not provided us with 
insights into the cost-benefit trade-offs and they have offered us quality as a magic bullet. 

We can agree that children’s child care experiences, like their other experiences, influence their 
development, though not in obvious or simple ways. Perhaps the next wave of research will enhance 
our understanding of how children’s experiences inside and outside their own homes combine to 
produce specific outcomes within different cultural and economic contexts. 



Abt Associates Inc. The Study of Cost, Quality and Child Outcomes in Child Care Centers 



21 



References 



Cost, Quality & Child Outcomes Study Team (1995). Cost, Quality, and Child Outcomes in Child 
Care Center, Public Report, second edition. Denver: Economics Department, University of 
Colorado at Denver. 

Helbum, S.W., (1995). Cost, Quality, and Child Outcomes in Child Care Centers, Technical Report, 
Denver, Department of Economics, Center for Research in Economic and Social Policy, 
University of Colorado at Denver. 

Lamb, M.E. (1997). Nonparental child care. Context, quality, correlates, and consequences. In I.E. 
Sigel, K.A. Renninger, & W. Damon (Eds.), Handbook of child psychology. Fifth Edition, 
Volume 4. New York Wiley. 

Peisner-Feinberg, E.S., Burchinal, M.R., Clifford, R.M., Culkin, M.L., Howes, C., Kagan, S. L., 
Yazejian, N., Byler, P., Rustici, J., & Zelazo, J. (2000). The children of the cost,, quality, and 
outcomes study go to school: Technical report. Chapel Hill: University of North Carolina at 
Chapel Hill, Frank Porter Graham Child Development Center. 



Abt Associates Inc. The Study of Cost, Quality and Child Outcomes in Child Care Centers 



22 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 





I. DOCUMENT IDENTIFICATION: 



Title: ^ ^ 




Author(s): 


Corporate Source; . ^ 

^ 0r 


Publication Date: 
2. oo<D 



II. REPRODUCTION RELEASE: 



In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, and ' 
electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if reproduction 
release is granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 



The sample sticker shown below will be 
affixed to all Level 1 documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 









TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




Check here for Level 1 r^ease, permitting reproduction 
and^fissemination in microfiche or otheTERlC archival 
media (e.g., electronic) and paper copy. 



The sample sticker shown below will be 
affixed to all Level 2A documents 



The sample sticker shown below will be 
affixed to all Level 2B documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY. 
HAS BEEN GRANTED BY 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



2A 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 






3 ^: 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



2B 



Level 2A 

t 



Level 2B 

t 



Check here for Level 2A release, permitting reproduction 
and dissemination in microfiche and in electronic media for 
ERIC archival collection subscribers only 



Check here for Level 2B release, permitting reproduction 
and dissemination in microfiche only 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce is granted, but no box is checked, documents will be processed at Level 1. 



gERiC 



Sign 

here, 

please 



/ hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this 
document as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and 
its system contractors requires permission from the copyright holder. Exception is made for non-profit reproductbn by libraries and other 


service agencies to satisfy information needs of educators in response to discrete inquiries. 




Signature/ y 


Printed Name/PositiorVTitle; 








Organization/Address; ^ 


Telephone; 


FAX 


P<(bT 


E-Mail Address; 

(jus&aJ,. 0 ,, 


t; 9. 2^.0 i 



(Over) 



i 



18L DOCUMENT AVAILAB8L1TY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless It is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 



Publisher/Distributor: 


Address: 




■ V-' • 












Price: 

















IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM: 



Karen E. Smith, Assistant Director 
Send this form to the following ERIC Clearinghouse: ERIC/EECE 

Children’s Research Center 
University of Illinois 
5 1 Gerty Dr. 

Champaign, IL 61820-7469 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 

ERIC Processing and Reference Facility 
4483-A Forbes Boulevard 
Lanham, Maryland 20706 

Telephone: 301-552-4200 
Toll Free: 800-799-3742 
FAX: 301-552-4700 
e-mail: ericfac@inet.ed.gov 
WWW: http://ericfaciiity.org 

fcl\lL088 (Rev. 2/2001) 



